[Bench] Add support for liboffload by 311Volt · Pull Request #22653 · intel/llvm

311Volt · 2026-07-16T12:39:48Z

Add support for liboffload in the benchmarking script in devops/scripts/benchmarks and its SubmitKernel implementation.

To help with local development, this PR also adds:

A --benchmarks-source-dir=<dir> option to point the Compute Benchmarks suite at an existing source tree instead of cloning intel/compute-benchmarks. When set, all git clone/fetch/checkout operations are skipped.
After a successful build, GitProject writes a JSON build-complete marker (benchmark_build_complete.json) into the build dir recording the source commit and git status --porcelain fingerprint. Reruns skip the build when the fingerprint still matches, so editing a tracked file rebuilds automatically. A non-git source dir always rebuilds.

These are opt-in, for local development only. When called without these options ('Run benchmarks' action), the script behaves as before.

Stacked on: #22640 . Review 35eecae and later.

In upstream LLVM, olCreateEvent gained a `Flags` param - update UR adapter to pass OL_EVENT_FLAGS_NONE, matching the old behavior. Concurrently, olLaunchKernel switched to a pointer array for the kernel launch args. Luckily the adapter already tracks the information needed for the new API, which can be passed directly. This patch is concerned with resolving build errors with current upstream LLVM and retaining existing functionality, not with feature-completeness against new liboffload features.

Make the benchmark runner usable for local development as opposed to only CI/CD, and add support for the new liboffload SubmitKernel benchmark. liboffload (OL) SubmitKernel: - Add an OL runtime, excluded from the default supported-runtime set so it stays scoped to SubmitKernel and does not leak into other benchmarks. - Add --offload-install-dir / --offload-include-dir (passed through as OFFLOAD_INSTALL_DIR / OFFLOAD_INCLUDE_DIR). When both are set, the suite builds with -DBUILD_OL=ON and enables the OL SubmitKernel benchmark. - Add --force-offload-plugin=<name> to export FORCE_OFFLOAD_PLUGIN (backend name, e.g. level_zero/cuda/amdgpu/host) into the benchmark process, which liboffload uses to select the offload device. Local development: - Add --benchmarks-source-dir=<dir> to point the Compute Benchmarks suite at an existing source tree instead of cloning intel/compute-benchmarks. When set, all git clone/fetch/checkout operations are skipped. - After a successful build, GitProject writes a JSON build-complete marker (benchmark_build_complete.json) into the build dir recording the source commit and `git status --porcelain` fingerprint. Reruns skip the build when the fingerprint still matches, so editing a tracked file rebuilds automatically. A non-git source dir always rebuilds.

dm-vodopyanov · 2026-07-16T13:32:06Z

@311Volt I see that you are a part of some internal Intel organizations on Github, so I suspect you are an Intel employee. I don't see your real name and Intel email in your profile. However, I see that you do commits with your Intel email address.

Because of lack of some info, we see the following:

@sarnex, could you please remind, what @311Volt should do so the workflows run automatically for him? One of the options is to complete the 1source onboard procedure (if was not completed yet).

EuphoricThinking · 2026-07-23T13:32:12Z

+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#   "matplotlib==3.9.2",
+#   "mpld3==0.5.10",
+#   "dataclasses-json==0.6.7",
+#   "PyYAML==6.0.1",
+#   "Mako==1.3.12",
+#   "psutil>=7.0.0",
+# ]
+# ///
+


There is already requirements.txt specifically for benchmarks. Even if the changes are intended just for the development, the inserted script affects the whole main.py which is also used for the release mode of benchmarks. Therefore, I can't see any intentional separation that would justify not appending the script content at the end of the linked requirements.txt. For the sake of consistency, those dependencies should be included in the existing file requirements.txt.

EuphoricThinking · 2026-07-23T13:38:55Z

@@ -4,6 +4,18 @@
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

+# /// script
+# requires-python = ">=3.10"


Why are newer versions required here? I am wondering whether this change would impact CI, for example, and force avoidable adjustments that would not bring many benefits for the affected components. Just hypothesizing.

EuphoricThinking · 2026-07-23T13:45:51Z

+    # Backend name (e.g. level_zero/cuda/amdgpu/host) exported as
+    # FORCE_OFFLOAD_PLUGIN for the benchmark executable process.


I was quite confused at the first reading. Maybe add a note about device selection from README.md? For me it sounds as if this variable was needed for all benchmarks, but the comment from README indicates that this is required only by OL benchmarks.

EuphoricThinking · 2026-07-23T13:49:36Z

Are these changes related to the main goal of this PR, to its scope declared in the description?

lukaszstolarczuk · 2026-07-24T07:18:53Z

+        default=options.offload_install_dir,
+    )
+    parser.add_argument(
+        "--offload-include-dir",


I only skimmed over the PR for now and these params got my attention - why do we have to specify 2 offload dirs? Can't you use a single path, like e.g. for UMF: --umf points to a prefix, where include and lib dirs are to be found.

lukaszstolarczuk · 2026-07-24T07:21:35Z

@@ -523,6 +535,34 @@ def validate_and_parse_env_args(env_args):
        help="Always download benchmark data dependencies, even if they already exist.",
        action="store_true",
    )
+    parser.add_argument(
+        "--benchmarks-source-dir",


I assume this option is more for development/debugging? We have env var: COMPUTE_BENCHMARKS_BUILD_PATH used for our testing of the framework - perhaps instead of adding new param here, use the env var instead...?

I wonder if you could just use the --offline option instead of adding a new one? Extending the usage of the env var is also a good idea.

lukaszstolarczuk · 2026-07-24T07:23:56Z

-        return super()._supported_runtimes() + [RUNTIMES.SYCL_PREVIEW]
+        runtimes = super()._supported_runtimes() + [RUNTIMES.SYCL_PREVIEW]
+        # OL (liboffload) SubmitKernel is only available when built via
+        # --offload-install-dir / --offload-include-dir.


nit: this comment is rather redundant, code seems self explanatory

lukaszstolarczuk · 2026-07-24T07:25:36Z

FYI, when you rebase, the benchmark CI issue will be gone.

PatKamin · 2026-07-24T07:25:42Z

        RUNTIMES.SYCL: "SYCL",
        RUNTIMES.LEVEL_ZERO: "Level Zero",
        RUNTIMES.UR: "Unified Runtime",
+        RUNTIMES.OL: "Offload",


Could be just RUNTIMES.OFFLOAD. More descriptive, still shorter than SYCL_PREVIEW or LEVEL_ZERO.

PatKamin · 2026-07-24T07:33:42Z

@@ -523,6 +535,34 @@ def validate_and_parse_env_args(env_args):
        help="Always download benchmark data dependencies, even if they already exist.",
        action="store_true",
    )
+    parser.add_argument(
+        "--benchmarks-source-dir",


I wonder if you could just use the --offline option instead of adding a new one? Extending the usage of the env var is also a good idea.

PatKamin · 2026-07-24T07:45:48Z


+# Marker file written into build_dir after a successful build. Records a
+# fingerprint of the source tree so reruns can skip rebuilding.
+BUILD_COMPLETE_MARKER = "benchmark_build_complete.json"


Why do we need this? All projects that are built are based on git so diffs are already easily checked.

PatKamin · 2026-07-24T07:48:19Z

+
+When a build finishes successfully, a JSON build-complete marker (`benchmark_build_complete.json`) is written into the project's build directory recording the source commit and `git status --porcelain` fingerprint. On subsequent runs the build is skipped when the fingerprint still matches, so editing a tracked source file triggers a rebuild automatically. If `--benchmarks-source-dir` points at a directory that is **not** a git repository, the benchmarks are always rebuilt.
+
+`--offload-install-dir <dir>` / `--offload-include-dir <dir>` - directories providing liboffload: the library dir containing `libLLVMOffload` (passed as `OFFLOAD_INSTALL_DIR`) and the header dir containing `OffloadAPI.h` (passed as `OFFLOAD_INCLUDE_DIR`). Both must be set together. When set, the OL (liboffload) `SubmitKernel` benchmark is built (via `-DBUILD_OL=ON`) and enabled. For an LLVM install tree built with `LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF` these are typically `<install>/lib` and `<install>/include/offload`.


This should be a single --offload path with subdirs derived from it.

311Volt added 2 commits July 6, 2026 10:56

311Volt requested review from a team as code owners July 16, 2026 12:39

run formatter

a9a2c1d

EuphoricThinking reviewed Jul 23, 2026

View reviewed changes

lukaszstolarczuk reviewed Jul 24, 2026

View reviewed changes

PatKamin requested changes Jul 24, 2026

View reviewed changes

		# Backend name (e.g. level_zero/cuda/amdgpu/host) exported as
		# FORCE_OFFLOAD_PLUGIN for the benchmark executable process.


		When a build finishes successfully, a JSON build-complete marker (`benchmark_build_complete.json`) is written into the project's build directory recording the source commit and `git status --porcelain` fingerprint. On subsequent runs the build is skipped when the fingerprint still matches, so editing a tracked source file triggers a rebuild automatically. If `--benchmarks-source-dir` points at a directory that is not a git repository, the benchmarks are always rebuilt.

		`--offload-install-dir <dir>` / `--offload-include-dir <dir>` - directories providing liboffload: the library dir containing `libLLVMOffload` (passed as `OFFLOAD_INSTALL_DIR`) and the header dir containing `OffloadAPI.h` (passed as `OFFLOAD_INCLUDE_DIR`). Both must be set together. When set, the OL (liboffload) `SubmitKernel` benchmark is built (via `-DBUILD_OL=ON`) and enabled. For an LLVM install tree built with `LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF` these are typically `<install>/lib` and `<install>/include/offload`.

Uh oh!

Conversation

311Volt commented Jul 16, 2026

Uh oh!

dm-vodopyanov commented Jul 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukaszstolarczuk commented Jul 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants